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OVERVIEW 


Given  the  current  emphasis  on  improving  all 
aspects  of  cost  analysis  and  the  primacy  of  cost 
uncertainty  in  the  cost  estimation  process,  it  is 
time  to  assess  the  quality  of  the  state-of-the-art 
and  look  toward  improving  techniques  where  needed. 

An  argument  supporting  the  primacy  of  cost 
uncertainty  analysis  and  a  critical  assessment  of 
existing  techniques  was  given .-in— {-U-fca.  In  this 
paper,  we  examine  some  efforts  to  apply  new 
results  and  insights,  both  mathematical  and 
psychological,  to  the  problems  of  cost  uncertainty 
analysis.  The  paper  will  be  treat  some  new  Ideas 
pertaining  to  probability  ellcltatlon/encodlng  using 
interactive  software  and  Indirect  assessment  of 
the  uncertainty  associated  with  a  CER. 

;/■  ^ 

1.  PROBABILITY  ELICITATION/ENCODING 

Even  In  the  presence  of  perfect  Information 
Vend  the  best  application  of  sound  cost  analysis 
techniques,  predicting  the  cost§  of,  say,  an 
apogee  kick  motor  ( AKM)  of  a  proposed  unmanned 
spacecraft  entails  unknown  quantities.  One  may 
attempt  a  bottoms-up  engineering  approach  listing 
design  costs,  material  and  labor  requirements, 
etc.,  or  use  a  cost  estimating  relation  (CER) 
developed  from  data  on  analogous  systems  and  their 
costs.  Whichever  approach  Is  taken,  perhaps  both, 
the  actual  cost  Is  not  predictable  with  certainty. 
Since  the  uncertainty  associated  with  AKM  cost 

I  The  term  "cost"  Is  used  In  a  generic  way  in 
this  paper.  Its  meaning  will  depend  on  the 
purpose  of  the  analysis  (e.g.  first  unit  cost, 
nonrecurring  cost,  life  cycle  cost,  etc.) 
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will  contribute  to  the  overall  system  cost 
uncertainty,  its  contribution  must  be  expressed  in 
some  analytically  useful  form,  generally  a 
probability  distribution.  Using  the  CER  approach, 
suppose  that  the  costa  of  AKMs  are  related  to 
total  impulse  by  the  simple  linear  model 

-  a  +  bCimpulse)^  + 

where  x  is  total  impulse  (measured  in  lb-sec),  $C 
is  standardized  cost,  a  and  b  are  unknown 
constants,  and  e  represents  variability  not 
accounted  for  by  x,  often  referred  to  an  error  or 
random  noise.  Thinking  of  the  present  system  and 
the  historical  systems  as  being  a  sample  from, 
say,  a  hypothetical  population  of  all  possible 
AKMs,  the  parameters  a,  b,  and  the  variance  of  the 
random  noise  can  be  estimated  in  the  usual  way. 

For  any  value  x*  of  impulse,  the  fitted  model 

$C  -  a  +  b<  X*  ) 

can  be  used  to  get  point  estimates  and  prediction 
Intervals  for  the  cost.  But  what  about  an 
uncertainty  distribution  over  the  cost?  Even  if 
normal  theory  applies,  the  predictive  distribution 
of  SC  is  only  partially  known.  It  depends  on  the 
unknown  values  of  a  and  b  and  on  the  unknown 
population  variance.  Even  if  these  values  were 
assumed  known,  as  they  are  in  [7]  and  [13],  there 
is  uncertainty  about  the  value  x*  of  the  input 
variable  x  (called  strategic  uncertainty  In  [1]). 
The  Impulse  of  the  finished  product  will  probably 
differ  from  Its  designed  nominal  value  since  the 
required  Impulse  will  depend  on  the  weight  of  the 
payload  about  which  there  Is  uncertainty.  These 
various  types  of  uncertainty  are  discussed  In  [11]. 
In  order  to  generate  an  uncertainty  distribution 
over  9C,  (called  a  "predictive  distribution"  since 
Its  purpose  Is  to  predict  $C),  one  must  quantify 
uncertainty  about  a,  b,  x,  and  the  random  noise 
variance.  Direct  probability  encoding  on  some  or 
all  of  these  quantities  is  a  difficult  cognitive 
task.  New  techniques  supported  by  Interactive 
computer  codes  have  been  developed  [9]  which  make 
these  tasks  easier.  They  will  be  discussed  In  the 
nsxt  section.  Shown  in  figure  l  Is  a  schematic 
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diagram  of  how  the  various  types  of  uncertainty 
come  together  to  generate  the  predictive 
distribution. 

FIGURE  1:  CER  Uncertainties 

MOOEL  ■  COST  -  *  )  +  6,  e-N(O.o’) 


COST  ANALYST  INPUTS  MATHEMATICAL  ANALYSIS 


To  be  specific,  what  Is  Involved  In  (direct) 
encoding  of  the  uncertainty  about  x,  the  Impulse? 
The  analyst  may  look  at  data  pertaining  to  the 
growth  of  Impulse  (l.e.  the  difference  between 
nominal  and  actual)  over  similar  AKMs.  He  may 
then  assess  a  few  key  quantities  from  which  a 
first  approximation  to  the  uncertainty 
distribution  Is  generated.  If  he  Is  using  the 
standard  PERT-beta  approach  (discussed  In  [11]) 
he  will  assess  a  lowest  possible  value  (1),  a  most 
likely  value  (a),  and  a  highest  possible  value 
(h).  A  fourth  assessment  relating  to  spread  (s) 
must  also  be  given.  Various  ways  for  doing  this 
are  summarized  In  [9].  This  process  can  be 
thought  of  as  specifying  a  few  conditions  (points 
or  properties)  on  the  cumulative  distribution 
function  (CDF).  For  example,  1  and  h  are  the  two 
extreme  points,  while  m  specifies  the  location  of 
the  Inflection  point.  Alternately,  a  few  central 
percentiles  might  be  specified  and  a  CDF  "faired" 


In  or  fitted  mathematically.  Behavioral  studies 
seem  to  support  this  latter  method  for  producing 
better  calibrated  assessments.  See  [6]  and  the 
cited  references  for  more  discussion  on  the 
Important  cognitive  issues  Involved.  Whatever 
method  is  used,  the  result  is  a  CDF  characterizing 
uncertainty  about  the  unknown  quantity. 

Generating  an  entire  CDF  based  on  a  few 
Inputs  involves  considerable  extrapolation.  The 
generated  CDF  is  supposed  to  embody  all  relevant 
information  including  experience-based  subjective 
judgment  of  the  cost  analyst.  He  should 
carefully  examine  the  implications  of  the  fitted 
CDF  to  Insure  its  fidelity  and  make  adjustments 
where  needed.  A  four-step  graphical  method  was 
proposed  in  [9]  for  accomplishing  this  type  of 
adjustment.  It  Involves  choosing  a  beta  shape 
from  a  collection  of  9  to  15  graphs.  Whenever  two 
consecutive  shapes  coincide,  the  analyst  is  termed 
consistent  and  the  process  stops.  The  suggested 
four-step  method  somewhat  resembles  an  earlier 
approach  (10]  which  was  Implemented  with  a 
prototype  interactive  computer  program.  Because 
of  its  popularity,  the  PBRT-beta  methodology  was 
chosen  for  this  implementation.  Software,  using 
the  more  versatile  and  applicable  Johnson  family 
of  distributions  discussed  below,  is  being 
developed.  Crude  pictures  of  successive  output 
screens  of  the  beta  code  are  shown  below.  User- 
supplied  Inputs  are  circled. 

In  the  first  screen,  the  user  is  prompted  for 
the  standard  PERT  input:  1,  a,  h.  A  beta  density 
with  the  PERT  variance  is  generated.  Some 
selected  percentiles  are  displayed  along  with  the 
beta  shape  parameters  and  other  descriptive 
information  (UNC  is  the  "UNcertalnty  Coefficient" 
defined  in  (4].  It  is  the  standard  deviation 
divided  by  that  of  a  uniform  distribution  over  the 
same  range.  Thus,  since  attention  is  limited  to 
unlmodal  betas,  UNC  ■  1  indicates  a  most  diffuse 
opinion.  (Smaller  values  indicate  less 
uncertainty.)  A  menu  of  available  modifications  is 
printed  and  the  user  is  prompted  to  select  an 
option. 


5 


FIGURE  2-1:  Initial  selection 
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The  output  of  the  initial  specification  is  shown 
above  along  with  the  menu.  Option  1  is  selected 
and  the  shift  value  entered  producing 

FIGURE  2-2 :  Result  of  shlftina  the  density 
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The  modified  (solid)  and  previous  (dash)  densities 
are  shown  in  figure  2-2.  Percentiles  and  other 
descriptive  information  is  listed  for  both 
densities  and  the  menu  repeated.  Option  2  (alter 
the  variance  but  not  the  range)  is  chosen  and  a 
"flattening"  of  10*  is  specified. 


FIGURE  2-3:  Result  of  flattening  the  density 
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The  flattening  effect  Is  evident  while  1,  m,  and 
h  are  unaltered.  Option  3  allows  one  to  set  any 
or  all  of  the  parameters.  Option  3  Is  selected  and 
alpha  Is  changed  to  2  resulting  In 


FIGURE  2-4:  Result  of  adjusting  parameter (s) 
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Alpha  has  been  changed  from  .92  to  2  thus  shifting 
the  mode  to  the  right. 


Option  4  allows  the  user  to  expand  or  contract  one 
tall  while  keeping  the  mode  and  other  tall  fixed. 
Here,  the  right  tall  Is  expanded  by  10*. 


FIGURE  2-5:  Result  of  adjusting  one  tall 
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The  user  can  recover  a  previous  density  by 
exercising  option  5.  This  saves  having  to  start 
all  over  If  a  mistake  Is  made  along  the  way. 
Here,  the  user  recovers  graph  n  *  2.  It  Is 
assigned  sequence  number  6  in 


FIGURE  2-6 
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Percentile  information  Is  valuable  for  checkin? 
consistency.  If  figure  2-1  accurately  captures 
the  analyst's  uncertainty  about  larger  costs,  the 
odds  against  the  cost  exceeding  27.378  are  3-1. 

If  the  analyst  likes  the  left  part  of  the  picture 
but  feels  that  3-1  odds  are  too  generous,  he  may 
use  option  4  to  lengthen  the  right  tall.  If  he  is 
uncomfortable  with  giving  or  taking  even  odds  that 
the  cost  will  not  exceed  22.192,  then  some  sort  of 
shift  may  be  indicated.  Including  percentile 
information  would  enhance  the  graphical  selection 
scheme  proposed  in  [9]. 

There  is  no  scientific  reason  for  using  beta 
distributions  (or  any  other,  for  that  matter)  in 
quantifying  cost  uncertainty.  Arguements  supporting 
other  choices  (e.g.  triangular  [14],  Welbull  [8]) 
are  usually  based  on  ease  of  computation.  In  this 
day  of  the  inexpensive  and  powerful  microcomputer, 
computational  ease  at  the  expense  of  flexibility 
of  application  seems  like  a  poor  tradeoff.  The 
main  criterion  for  Judging  the  suitability  of  a 
family  of  densities  should  be  its  richness  of 
possible  shapes.  A  particularly  nice  shape-rich 
family  which  seems  to  have  been  overlooked  was 
proposed  by  N.  L.  Johnson  [3]  and  bears  his  name. 

A  clear  account  is  given  in  the  text  [2].  The 
family  is  much  richer  in  shape  possibilities  than 
the  beta  family.  Its  skewness  (beta  1)  and 
kurtosls  (beta  2)  cover  the  entire  Pearson  plane 
shown  in  figure  3  with  subfamily  Sb  covering  the 
region  above  the  lognormal  line  and  subfamily  Su 
covering  the  region  below.  The  lognormal  family 
belongs  to  the  Johnson  family  and  is  denoted  SI. 

By  contrast,  Welbull  and  triangular  shape 
parameters  are  resrlcted  to  the  curves  shown. 

A  particularly  nice  feature  of  the  Johnson  family 
is  that  its  percentiles  can  be  calculated  using  a 
table  of  standard  normal  probabilities.  This 
facilitates  Monte  Carlo  simulation  work.  Research 
on  fitting  these  distributions  to  subjective 
assessments  is  underway  [12]  and  preliminary 
results  are  encouraging. 


FIGURE  3:  Skewness  and  Kurtosis  of  Several 
Families  of  Densities 


TRAJECTORIES  IN  PEARSON  PLANE 


2.  CER  UNCERTAINTY 

In  Section  1,  three  sources  of  uncertainty 
which  contribute  to  overall  CER  uncertainty  were 
identified.  The  task  of  quantifying  opinion  about 
model  parameters  such  as  a  and  b  is  necessary  in 
order  to  generate  an  uncertainty  distribution  over 
the  subsystem  cost.  Direct  assessment  of  a  joint 
distribution  is  particularly  difficult.  In  a 
recent  paper  [5],  a  technique  has  been  developed 
to  indirectly  assess  this  distribution.  Opinions 
expressed  by  the  analyst  are  treated  as  data  to 
estimate  the  model  parameters.  By  asking 
questions  about  the  predictive  distribution  at 
various  values  of  the  Independent  variables,  it 
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can  be  inferred  what  the  prior  distribution  has  to 
be  in  order  to  be  consistent  with  these 
assessments.  The  elicited  assessments  are  in  the 
form  of  quantile  of  the  predictive  distribution. 
This  approach  was  taken  in  the  belief  that 
quantiles  are  "less  subject  to  erratic  behavior, 
easier  to  think  about,  and  consequently  easier  to 
elicit."  Consistency  checks  are  made  in  the 
process  so  that  answers  that  appear  to  be  out  of 
line  with  other  answers  can  be  called  to  the 
attention  of  the  user.  The  techniques  have  been 
Implemented  with  an  interactive  program  on  the 
TROLL  experimental  system  operated  by  MIT.  With 
proper  training,  this  new  approach  may  become  a 
useful  tool  in  cost  uncertainty  analysis. 
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